Papers with in-depth examination
Is This LLM Library Learning? Evaluation Must Account For Compute and Behaviour (2026.eacl-long)
Copied to clipboard
| Challenge: | Recent advances in the coding, reasoning, and tool-use ability of LLMs have raised the possibility of library learning with LLM. |
| Approach: | They propose to use reusable and composable functions and tools to create reusable, composesable code and tools that can be reused by modifying relevant examples. |
| Outcome: | The proposed system fails to consistently outperform the baseline model and does not correct for the difference in computational cost. |
DecipherPref: Analyzing Influential Factors in Human Preference Judgments via GPT-4 (2023.emnlp-main)
Copied to clipboard
| Challenge: | Human preference judgments are important in large language models to produce outputs that align with human values. |
| Approach: | They conduct an in-depth examination of pairwise human judgments released by OpenAI . they find that most favored factors vary across tasks and genres . |
| Outcome: | The proposed model reveals that most favored factors vary across tasks and genres . the findings have implications on the construction of balanced datasets in human preference evaluations - crucial step in shaping behavior of future LLMs. |
EpLSA: Synergy of Expert-prefix Mixtures and Task-Oriented Latent Space Adaptation for Diverse Generative Reasoning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing models for diverse generative reasoning struggle to generate multiple unique and plausible results. |
| Approach: | They propose a model based on expert-prefix mixtures and task-oriented latent space adaptation for diverse generative reasoning. |
| Outcome: | The proposed model outperforms existing models on three types of generative reasoning tasks. |
Enhancing Data Quality through Simple De-duplication: Navigating Responsible Computational Social Science Research (2024.emnlp-main)
Copied to clipboard
| Challenge: | Social media data exhibits distinctive characteristics such as rapid and continual topic evolution. |
| Approach: | They propose new protocols and best practices for improving dataset development from social media data and its usage. |
| Outcome: | The proposed protocols and best practices improve the performance of social media datasets and their usage. |
Revisiting Automated Evaluation for Long-form Table Question Answering (2024.emnlp-main)
Copied to clipboard
| Challenge: | Existing automated metrics for long-form table question answering (LFTQA) are poorly correlated with human judgments and fail to distinguish between factually accurate responses and those that are factual incorrect. |
| Approach: | They propose to use a meta-evaluation dataset to assess the effectiveness of LLM-based LFTQA systems. |
| Outcome: | The proposed meta-evaluation dataset includes 2,988 human-annotated examples. |